Supervised Learning:
Classification

What is classification?

Regression (which you will meet in the next lesson) predicts a number on a continuous scale. Classification does something different. It sorts an input into a discrete bucket, one of a fixed set of categories. You are not predicting how much, you are predicting which one.

Think about your email inbox. Every time a new message arrives, something has already decided whether it belongs in your main inbox or your spam folder. It looked at the sender, the subject, the links, the phrasing, and placed the email into one of two buckets. That is a classifier at work, running thousands of times per second across every inbox on the planet.

For classification to work, you need labelled data. That means historical examples where someone or something has already recorded the correct category. A spam filter trained on emails that humans previously marked as spam or not spam. A medical model trained on scans that doctors already diagnosed. No labels, no classification.

Decision Trees: the if-then machine

The decision tree is one of the most intuitive machine learning algorithms. It mirrors the way humans naturally reason through choices: "If this, then that. Otherwise, check the next thing." You can trace exactly why a decision tree made any particular decision, which makes it one of the most explainable models in existence.

A decision tree learns a sequence of questions from your training data. At each branch, it asks a question about one feature. Depending on the answer, it goes left or right. It keeps branching until it reaches a leaf, which holds a class label. The tree learns which questions to ask and in which order by finding the split at each step that best separates the classes.

The strength of decision trees is their readability. A bank can use a decision tree for loan approvals and explain to any rejected applicant exactly why they were turned down, step by step. That matters a lot in regulated industries.

The weakness is overfitting. A tree that is allowed to grow freely will memorise the training data perfectly, asking increasingly specific questions until it has a unique path for every single training example. On new data, it falls apart. This is fixed by limiting depth, or by using an ensemble of many trees, which is what Random Forests do.

K-Nearest Neighbours: birds of a feather

K-Nearest Neighbours (KNN) is probably the most intuitive classification algorithm in existence. The idea is almost embarrassingly simple. To classify a new data point, look at the K closest training examples in the dataset. Whatever class appears most often among those neighbours, assign that class to the new point.

KNN requires no training in the traditional sense. It simply stores all the training data. When a new point needs to be classified, it calculates the distance to every stored point, finds the K closest ones, and takes a vote. Simple, effective, and surprisingly powerful on many real problems.

The catch is that KNN gets slow as your dataset grows. With millions of training examples, computing the distance to every single one for each prediction becomes expensive. It also struggles with high-dimensional data where "closeness" becomes a strange and unreliable concept. Still, for small to medium datasets, KNN is a rock-solid baseline and a great first thing to try.

Building a classifier with Scikit-learn

This is where theory becomes code. Scikit-learn makes building a classifier a matter of a few lines. The pattern is always the same: load and prepare data, split into train and test sets, create the model, fit it, predict, evaluate.

import pandas as pd
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, classification_report

# Load and prepare the Titanic dataset
df = pd.read_csv('titanic.csv')

# Select features and target
features = ['Pclass', 'Age', 'SibSp', 'Fare']
df = df[features + ['Survived']].dropna()

X = df[features]
y = df['Survived']

# Split: 80% train, 20% test
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Build and train the decision tree
model = DecisionTreeClassifier(max_depth=4, random_state=42)
model.fit(X_train, y_train)

# Make predictions on the unseen test set
predictions = model.predict(X_test)

# Evaluate
print("Accuracy:", accuracy_score(y_test, predictions))
print(classification_report(y_test, predictions))

About 80% of the time the model correctly predicts whether a passenger survived, based only on their ticket class, age, number of siblings or spouses aboard, and fare paid. Not bad for a tree with four levels of depth.

Notice max_depth=4 in the code. That is a hyperparameter you chose before training. Change it to 20 and the accuracy on training data goes up, but the test accuracy drops as the model starts memorising instead of generalising.

Reading a confusion matrix

Accuracy alone can mislead you. Imagine a model trained to detect a disease that affects 1% of the population. A model that simply predicts "no disease" for every single person achieves 99% accuracy. But it misses every actual sick person. That is a useless model.

The confusion matrix breaks down your model's mistakes in a way that tells a much richer story. It shows you not just how often the model was right, but which kinds of errors it made.

The green cells are correct predictions. The red cells are errors. You can see the model makes more false negatives (20 people it predicted would die but actually survived) than false positives (14 people it predicted would survive but actually died).

Whether that matters depends on the context. In medical diagnosis, a false negative (missing a real cancer case) is often far more dangerous than a false positive (unnecessary further testing). In spam filtering, a false positive (deleting a real email) is more annoying than a false negative (letting spam through). The confusion matrix helps you see which error type your model is prone to, so you can tune it for the situation.